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DETAILED ACTION 

Response to Amendment 

1 . This action is responsive to applicant's amendment and remarks received on 8/28/08. 
Claims 1-8 are currently pending. 

Claim Rejections - 35 USC §102 

2. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this counliy, or patented or described in a printed publication in tliis 
or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale m this country, more than one year prior to the date of application for patent m the United States. 

(e) the invention was described in (1) an application for patent, published imder section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

3. Claims 1, 2, 3/1 are rejected under 35 U.S.C. 102(b) as being anticipated by Michael et al 

(US 6,421,458 62). 

Regarding claim 1, Michael discloses an image processing device for determining three- 
dimensional position and/or orientation of an object, comprising: 

image data capturing means for capturing image data containing an image of the object 
(see figure la, numeral 100, col. 4, lines 31-45, imaging system 100); 
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reference model pattern creating means for creating a reference model pattern based on 
image data of a reference object with a three-dimensional reference orientation relative to said 
image capturing means captured by said image capturing means, said reference object having a 
shape substantially identical to that of the object (see fig. 2a, numeral 200, col. 4, lines 31-44; 
series of training images are captured of an object or plurality of objects 104, the same object 
may be presented at a range of positions and orientations relative to the imaging system 100, the 
training objects 104 comprise objects known to be representative samples so as to produce the 
most accurate statistics); 

transformation means for performing two-dimensional and geometrical transformation on 
the created model pattern using a plurality of parameter sets to generate a transformed model 
pattern representing an image of the object with three-dimensional orientation different from the 
reference orientation (see fig. 2a, numeral 210, col. 4, lines 59-67, col. 5, lines 1-65; alignment 
model image 1 14 and each training image 101 are presented to an alignment system 106 for 
determining the affine pose 107 of each training image with respect to the alignment model 
image and uses the affine pose 107 comprises a set of parameters which describe how the 
training image can be transformed mathematically so as to align the training image with the 
alignment model image, each filtered training image 109 and its corresponding affine pose 
parameters 107 are applied to the General Affine Transform 1 10 to generate transformed training 
images 111, following transformation, a template image 1 13 is computed in step 212); 

pattern matching means for performing a pattern matching of the image data of the object 
captured by said image capturing means with the transformed model pattern (see fig. 2b, numeral 
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234, col. 6, lines 35-67, col. 7, lines 1-8; generate a transformed image 311 which align 
substantially with the template images 1 13); 

selecting means for repeatedly performing the generation of a transformed model pattern 
(see fig. 2a, numeral 216; col. 6, lines 8-20; a determination is made as to whether training is 
complete, if not additional training images may be captured) and the pattern matching of the 
image data of the object with the transformed model pattem to thereby select one of the 
transformed model patterns in conformity with the image data of the object, and obtain 
information on a position of the image of the object in the image data (see fig. 2b, numeral 234, 
col. 3, lines 5-10, col. 6, lines 35-67, col. 7, lines 1-8; process of transforming the filtered 
training and run-time images implies that the run-time process is executed more than once; 
generate a transformed image 311 which align substantially with the template images 1 13); 

means for obtaining information on a position of the image of the object in accordance 
with the selected one of the transformed model pattems in the image data (see col. 3, lines 5-10, 
col. 6, lines 35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled 
regions are computed); 

means for obtaining information on the three-dimensional orientation of the object based 
on one of the parameter sets used for generating the selected one of the transformed model 
pattems (see col. 4, lines 7-18, col. 4, lines 66-67; col. 5, lines 1-17; affine parameters apply to 
six degrees of freedom to compensate for image scale, shear, rotation, skew, and translation 
assuming a two-dimensional image of a three-dimensional object); and 

determining means for determining three-dimensional position and/or orientation of the 
object based on the information on the position of the image of the object in the image data and 



Application/Control Number: 10/807,259 Page 5 

Art Unit: 2624 

information on the three-dimensional orientation of the object (see col. 3, lines 5-10, col. 6, lines 
35-67, col. 7, lines 1-8, col. 5, lines 1-17; statistics on the area, position, and orientation of the 
labeled regions are computed; two-dimensional image of a three-dimensional object, the affine 
parameters apply to six degrees of freedom, to compensate for image scale, shear, rotation, skew 
and translation). 

Regarding claim 2, Michael discloses an image processing device for determining three- 
dimensional position and/or orientation of an object, comprising: 

image data capturing means for capturing image data containing an image of the object 
(see figure la, numeral ICQ, col. 4, lines 31-45, imaging system 100); 

reference model pattern creating means for creating a reference model pattern based on 
image data of a reference object with a three-dimensional reference orientation relative to said 
image capturing means captured by said image capturing means, said reference object having a 
shape substantially identical to that of the object (see fig. 2a, numeral 200, col. 4, lines 31-44; 
series of training images are captured of an object or plurality of objects 104, the same object 
may be presented at a range of positions and orientations relative to the imaging system 100, the 
training objects 104 comprise objects known to be representative samples so as to produce the 
most accurate statistics); 

transformation means for performing two-dimensional and geometrical transformation on 
the created model pattern using a plurality of parameter sets to generate a transformed model 
pattern representing an image of the object with three-dimensional orientation different from the 
reference orientation (see fig. 2a, numeral 210, col. 4, lines 59-67, col. 5, lines 1-65; alignment 
model image 114 and each training image 101 are presented to an alignment system 106 for 
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determining the affine pose 107 of each training image with respect to the alignment model 
image and uses the affine pose 107 comprises a set of parameters which describe how the 
training image can be transformed mathematically so as to align the training image the alignment 
model image, each filtered training image 109 and its corresponding affine pose parameters 107 
are applied to the General Affine Transform 1 10 to generate transformed training images 111, 
following transformation, a template image 1 13 is computed in step 212); 

storage means for storing the plurality of transformed model patterns and the parameter 
sets used in generating the respective transformed model patterns to be associated therewith (see 
figure la, numeral 1 12, col. 5, lines 39-53; transformed training images are preferably stored in a 
pair of accumulators 1 12); 

pattern matching means for performing pattem matching of the image data of the object 
captured by said image capturing means with the plurality of transformed model patterns to 
thereby select one of the transformed model patterns in conformity with the image data of the 
object (see fig. 2b, nimieral 234, col. 6, lines 35-67, col. 7, lines 1-8; generate a transformed 
image 311 which align substantially with the template images 1 13); and 

means for obtaining information on a position of the image of the object in accordance 
with the selected one of the transformed model pattems in the image data (see col. 3, lines 5-10, 
col. 6, lines 35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled 
regions are computed); 

means for obtaining information on the three-dimensional orientation of the object based 
on one of the parameter sets used for generating the selected one of the transformed model 
pattems (see col. 4, lines 7-18, col. 4, lines 66-67; col. 5, lines 1-17; affine parameters apply to 
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six degrees of freedom to compensate for image scale, shear, rotation, skew, and translation 
assuming a two-dimensional image of a three-dimensional object); and 

determining means for determining three-dimensional position and/or orientation of the 
object based on the information on the position of the image of the object in the image data and 
information on the three-dimensional orientation of the object (see col. 3, lines 5-10, col. 6, lines 
35-67, col. 7, lines 1-8, col. 5, lines 1-17; statistics on the area, position, and orientation of the 
labeled regions are computed; two-dimensional image of a three-dimensional object, the affine 
parameters apply to six degrees of freedom, to compensate for image scale, shear, rotation, skew 
and translation). 

Regarding claim 3/1, Michael teaches a two-dimensional and geometrical transformation 
of an affine transformation (see col. 4, lines 66-67; col. 5, lines 1-17), and said image processing 
device further comprises additional measuring means for obtaining a sign of inclination of the 
object with respect to said image capturing means (see col. 4, lines 66-67; col. 5, lines 1-17). 

Claim Rejections - 35 USC § 103 
4. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



5. Claims 7 and 8 are rejected under 35 U.S. C. 103(a) as being unpatentable over the 
combination of Michael et al (US 6,421,458 B2) in view of Watanabe et al (EP 1043689 A2). 
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Regarding claims 7, 8, Michael discloses all elements as mentioned above in claim 1 . 
Michael does not disclose a storage means storing an operating orientation of the robot relative 
to the object or storing an operating orientation and an operating position of the robot relative to 
the object; and robot control means for determining an operating orientation of the robot or the 
operating orientation and an operating position of the robot based on the determined three- 
dimensional position and/or orientation of the object; and an image capturing means mounted on 
the robot. 

Watanabe, in the same field of endeavor, teaches a storage means storing an operating 
orientation of the robot relative to the object or storing an operating orientation and an operating 
position of the robot relative to the object ("when a picking-up command is inputted .... three 
dimensional position/posture of the camera 20 on the world coordinate system at this image 
capturing position is outputted to the image processing apparatus 30"; Michael: paragraph 
[0040]); and robot control means for determining an operating orientation of the robot or the 
operating orientation and an operating position of the robot based on the determined three- 
dimensional position and/or orientation of the object ("robot controller 10 operates the robot to 
perform a pick-up operation .... based on the three-dimensional position/posture of the 
workpiece W (Step 210)"; Michael: paragraph [0044]); and an image capturing means mounted 
on the robot (Michael: figure 1, numeral 20). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference, to be incorporated into a robot system as suggested by 
Watanabe, to enhance inspection process by "demanding relatively less on system storage, and 
improv[ing] system speed and accuracy" (Michael: col. 2, lines 5-12). 
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6. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over the combination 
of Michael et al (US 6,421,458 B2) with Maeda et al (US 2003/0161537 Al), and further in view 
of Inoue (US 2003/0161504 Al). 

Regarding claim 4, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach dividing of a model pattern into at least two partial model patterns which 
are subject to the affine transformation to generate transformed partial model patterns, and 
pattern matching of the image data of the object with the transformed partial model pattems to 
determine most conformable sizes, and determines the sign of the inclination based on 
comparison of the sizes of the conformable partial model pattems with each other. 

Maeda teaches a pattern matching of the image data of the object with the transformed 
model pattems to determine most conformable sizes, and determines the sign of the inclination 
based on comparison of the sizes of the conformable model pattems with each other (Maeda: 
figure 1, numeral 32). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference to determine the sign of the inclination based on 
comparison of the sizes of the conformable partial model pattems with each other as suggested 

by Maeda, to allow the imaging device to accurately identify "a three-dimensional object using 
images obtained by photographing the object in the various directions" (Maeda: paragraph 
[0004]). 

The Michael with Maeda combination as applied above does not teach dividing of a 
model pattem into at least two partial model pattems which are subject to the affine 
transformation to generate transformed partial model pattems. 
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Inoue teaches dividing of a model pattern into at least two partial model patterns (Inoue: 
figure 2, numeral 302, 303), which are subject to the affine transformation ("affine 
transformation"; Inoue: paragraph [0107]) to generate transformed partial model patterns. 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael with Maeda combination as applied above, to utilize partial model 
patterns as suggested by Inoue, to further "correctly classify the input image regardless of a 
fluctuation in illumination, and a state of occlusion" (Inoue: paragraph [0012]). 
7. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Michael et al (US 
6,421,458 B2) in view of Okisu et al (US 6,806,903 Bl). 

Regarding claim 5, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach a measurement of distances from a displacement sensor separately 
provided in the vicinity of said image capturing means to at least two points on the object using 
the displacement sensor, and determines the sign of the inclination based on comparison of the 
measured distances. 

Okisu teaches a measurement of distances from a displacement sensor separately 
provided in the vicinity of said image capturing means to at least two points on the object using 
the displacement sensor, and determines the sign of the inclination based on comparison of the 
measured distances ("measure distance to two separate portions of an object from the electronic 
camera, and calculate an angle of inclination 6 based on measure two distances"; Okisu: col. 19, 
lines 54-57). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference to determine the sign of the inclination based on 
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comparison of the measured distances as suggested by Okisu, to allow the image processing 
device to properly focus with high degree of accuracy on the targeted object for image capturing. 
8. Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over Michael et al (US 
6,421,458 B2) with Watanabe et al (EP 1043689 A2), and further in view of Maeda et al (US 
2003/0161537 Al). 

Regarding claim 6, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach additional pattern matching of image data of the object captured after 
said image data capturing means is slightly moved or inclined and determining the sign of the 
inclination based on judgment whether an inclination of image of the object becomes larger or 
smaller than the selected one of the transformed model patterns. 

Watanabe, in the same field of endeavor, teaches an additional pattern matching of image 
data of the object captured after said image data capturing means is slightly moved or inclined 
("camera may be moved parallelly in accordance with the position of the workpiece in the field 
of view of the camera"; Watanabe: paragraph: [0048]). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference to move the image data capturing means as suggested by 
Watanabe, to provide more training images to the classifier in order for a more robust collection 
of data to enhance the classifier for increased reliability of the system. 

Maeda teaches additional pattern matching of image data of the object captured after said 
image data capturing means is slightly moved or inclined (see fig. 4) and determining the sign of 
the inclination based on judgment whether an inclination of image of the object becomes larger 
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or smaller than the selected one of the transformed model patterns (Maeda: figure 1, numeral 
32). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael with Watanabe combination to determine the sign of the inclination 
based on judgment whether an inclination of image of the object becomes larger or smaller than 
the selected one of the transformed model patterns as suggested by Maeda, to allow the imaging 
device to accurately identify "a three-dimensional object using images obtained by 
photographing the object in the various directions" (Maeda: paragraph [0004]). 



Response to Arguments 

9. Applicant's arguments filed 8/28/08, in regards to claim 1, have been fiiUy considered but 
they are not persuasive. Applicant argues that the Michael reference does not disclose 
performing 2-D and geometrical transformation on the crated model pattern to generate a 
transformed model pattern representing an image of the object with a 3-D orientation different 
from the reference orientation (see pg. 6, second paragraph). This argument is not considered 
persuasive since Michael discloses within fig. 2a, numeral 210, col. 4, lines 59-67, col. 5, lines 1- 
65; alignment model image 114 and each training image 101 are presented to an alignment 
system 106 for determining the affine pose 107 of each training image with respect to the 
alignment model image and uses the affine pose 107 comprises a set of parameters which 
describe how the training image can be transformed mathematically so as to aUgn the training 
image with the alignment model image, each filtered training image 109 and its corresponding 
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affine pose parameters 107 are applied to the General Affine Transform 1 10 to generate 
transformed training images 111, following transformation, a template image 1 13 is computed in 
step 212. Therefore, Michael clearly discloses that the training image is transformed that has a 
3-D orientation different from the reference orientation. 

Applicant argues that the Michael reference has the training image and the aUgnment 
model image is not transformed in order to align with the alignment model image (see pg. 6, 
third paragraph). This argument is not considered persuasive since the Examiner acknowledges 
that the training image is transformed, but the training image is considered the reference model 
pattern which is transformed into a transformed model pattern that is different from the reference 
object pattern. 

Applicant argues that the Michael reference does disclose pattern matching of the image 
data of the object with a transformed model pattern that is generated to represent an image of the 
object different from the reference orientation (see pg. 6, fourth paragraph). This argument is not 
considered persuasive since alignment of the template and threshold images is considered pattern 
matching since in order to aUgn two objects a matching must be executed to displace the objects 
in an appropriate position/orientation. 

Applicant argues that the Michael reference does not disclose repeatedly performing the 
generation of a transformed model pattern and the pattern matching of the image data of the 
object with the transformed model pattem to thereby select one of the transformed model 
patterns in conformity with the image data of the object (see pg. 6, fifth paragraph). This 
argument is not considered persuasive since Michael discloses the cited limitation in fig. 2a, 
numeral 216; col. 6, lines 8-20; a determination is made as to whether training is complete, if not 
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additional training images may be captured and fig. 2b, numeral 234, col. 3, lines 5-10, col. 6, 
lines 35-67, col. 7, lines 1-8; process of transforming the filtered training and run-time images 
implies that the run-time process is executed more than once; generate a transformed image 311 
which align substantially with the template images 113. The cited Michael reference is 
equivalent to the cited limitation as seen above in the argument and rejection. 

Applicant argues that the Michael reference does not disclose information on a position 
of the image of the object in accordance with a selected transformed model pattern in obtained or 
that information on the 3-D orientation of the object based on one of the parameter sets used for 
generating the selected transformed model pattern is obtained (see pg. 6, last paragraph, pg. 7, 
first paragraph). This argument is not considered persuasive since the Michael reference 
discloses the cited limitations within col. 3, lines 5-10, col. 6, lines 35-67, col. 7, lines 1-8; 
statistics on the area, position, and orientation of the labeled regions are computed and col. 4, 
lines 7-18, col. 4, lines 66-67; col. 5, lines 1-17; affine parameters apply to six degrees of 
freedom to compensate for image scale, shear, rotation, skew, and translation assuming a two- 
dimensional image of a three-dimensional object. The cited section is equivalent to the cited 
claim limitation since the limitation only brings in the concept of obtaining information on a 
position of the image of the object based on the transformed model pattern, which is clearly 
shown when an error image that is based off of the transformed model pattern produces 
information on the area, position and orientation of the labeled regions which are further 
computed to classify the object. The second concept is having information on the orientation of 
the object based on one of the parameter sets, not on the actual transformed model pattem, which 
is clearly disclosed by the cited section. 
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Applicant argues that Michael does not disclose determining 3-D position and/or 
orientation of the object based on the information on the position of the image of the object and 
the 3-D orientation of the object. This argument is not considered persuasive since the cited 
limitation only calls for determining 3-D position which is disclosed within col. 3, lines 5-10, 
col. 6, lines 35-67, col. 7, lines 1-8, col. 5, lines 1-17; statistics on the area, position, and 
orientation of the labeled regions are computed; two-dimensional image of a three-dimensional 
object, the affine parameters apply to six degrees of freedom, to compensate for image scale, 
shear, rotation, skew and franslation. 

Applicant argues cited limitations within claim 1 are not disclosed by the Michael 
reference (see pg. 7, second paragraph - pg. 8, first paragraph). This argument is not considered 
persuasive since the arguments have been addressed for the cited limitations as seen above in the 
argument section. 

Regarding claim 3/1, applicant argues that the claim is allowable due to the dependency 
from claim 1 (see pg. 8, second paragraph). This argument is not considered persuasive since the 
claim 1 stands rejected and the arguments and rejection can be seen above. AppUcant argues that 
Michael does not disclose the limitations of claim 3. This argument is not considered persuasive 
since the cited limitation is taught and can be seen above in the rejection of claim 3/1 . 

Regarding claims 4-8, applicant argues that the claims are allowable due to the 
dependency from claim 1 (see pg. 8, last paragraph). This argument is not considered persuasive 
since the rejection of claim 1 stands and the arguments and rejection can be seen above. 
Furthermore, apphcant argues that the combination does not disclose claim 4. This argument is 
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not considered persuasive since the combination of Michael, Maeda, with Inoue disclose all the 
limitations as cited in claim 4 and can be seen above in the rejection of claim 4. 



Conclusion 

10. THIS ACTION IS MADE FINAL. Apphcant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to EDWARD PARK whose telephone number is (571)270-1576. 
The examiner can normally be reached on M-F 10:30 - 20:00, (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on (571) 272-7415. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Edward Park 
Examiner 
Art Unit 2624 

/Edward Park/ 
Examiner, Art Unit 2624 



/Brian P. Werner/ 

Supervisory Patent Examiner, Art Unit 2624 



